Reproducibility in Actuarial Workflows

Introduction

• We are writing more code
• Need to be able to reproduce results from past projects, for review or audit purposes
• We need tools to:
  • Write
  • Collaborate (not covered today but mentioned)
  • Review
  • Reproduce projects
• Software engineers face the same challenges and solved them decades ago

What do we mean by reproducibility? And why should you care?

Getting the same results from an analysis when the same data is used

Examples (1/2)

• Famous Paper: Reinhart and Rogoff
• Published a paper that showed that when countries are too much in debt (over 60% of GDP according to the authors) then annual growth decreases by two percent.
• Serious problem: Performed calculations in excel and did not (this actually happened) select every country’s real GDP growth to compute average real GDP growth for high debt countries by governments.

Examples (1/2)

Examples (1/2)

• This paper was used to justify austerity measures for European countries during the 2009 crisis

Examples (2/2)

• Within the department:
  1. Bancassurance code
  2. All excel files - one click away from messing up your day
  3. Risk Integrity Data files

Levels of reproducibilty (1/2)

Here are the five main things influencing an analysis’ reproducibility:

• Version of language used
• Version of packages used
• Operating system
• Hardware (basically your computer)
• Documentation

Levels of reproducibility (2/2)

Source: Peng, Roger D. 2011. “Reproducible Research in Computational Science.” Science 334 (6060): 1226–27

Reproducibility Iceberg

Risks to mitigate: Language versions (1/2)

R < 3.6 (set.seed(1234))

sample(seq(1, 10), 5)

[1] 2 6 5 8 9

sample(seq(1, 10), 5)

[1] 10  6  5  4  1

Risks to mitigate: Language versions (2/2)

Python >= 3.6+

name = "Alice"
print(f"Hello, {name}!")

Hello, Alice!

name = "Alice"
print(f"Hello, {name}!")

File "example.py", line 2
    print(f"Hello, {name}!")
                          ^
SyntaxError: invalid syntax

Risk to mitigate: Package Versions (1/2)

{stringr} < 1.5.0:

stringr::str_subset(c("", "a"), "")

[1] "a"

stringr::str_subset(c("", "a"), "")

Error in `stringr::str_subset()`:
! `pattern` can't be the empty string (`""`).
Run `rlang::last_trace()` to see where the error occurred.

Risk to mitigate: Package Versions (2/2)

NumPy < Prior 2008

import numpy as np
np.array([1]) / 2

0

import numpy as np
np.array([1]) / 2

0.5

Risks to mitigate: Operating systems (1/3)

Rarely an issue, but Neupane, et al. 2019:

While preparing a manuscript, to our surprise, attempts by team members to replicate these results produced different calculated NMR chemical shifts despite using the same Gaussian files and the same procedure outlined by Willoughby et al. […] these conclusions were based on chemical shifts that appeared to depend on the computer system on which step 15 of that protocol was performed.

Risks to mitigate: Operating systems (2/3)

Risks to mitigate: Operating systems (3/3)

The problem

Works on my machine!

Project start

• Our project: Insurance Service Result
• Data to analyse: IRA Report 2023
• 2 scripts to analyse the data:

Project start: What’s wrong with these scripts?

• The first two scripts -> script-based workflow
• Just a long series of calls
• No functions meaning:
  • difficult to re-use!
  • difficult to test!
  • difficult to parallelise!
  • lots of repetitions (plots)
• Usually we want a report not just a script; in this case a dashboard

Turning our scripts reproducible

We need to answer these questions

The easiest, cheapest thing you should do

• Write an audit trail/documentation
• Generate a list of used packages and programming language:
  • In R, we can use {renv}
  • In Python, we can use venv/pipenv/conda/uv/poetry/virtualenv
• You can see how much of an issue this is in Python with all the tools available.

Restoring a library

Shortcomings

Where are we on the reproducibility iceberg?

• Package and language versions are recorded
• Packages can be restored (not 100% chance of always working, but certainly not 0%)
• You could stop here, I think this is enough
• But let’s go one step further (for now), let’s create a pipeline

Build automation using {targets} - R only

• Anatomy of the _targets.R script
• What’s a “target”?
• Dependencies of the pipeline?
• How to inspect the pipeline?
• How to run the pipeline?
• How to inspect a computed target?

Only outdated targets get recomputed

• Inspect the pipeline using targets::tar_visnetwork()
• Let’s remove a company from the list of companies
• Inspect the pipeline again
• Rerun the pipeline! Only what’s needed gets recomputed

Our analysis as a pipeline (1/3)

• [Discussion] What’s the difference between this pipeline and our original scripts?
• Benefits:
  1. _targets.R provides a clear view of what’s happening (beyond documentation)
  2. Functions can be easily re-used (and packaged!)
  3. The pipeline is pure (the results don’t depend on any extra, manual, manipulation!)

Our analysis as a pipeline (2/3)

4. The pipeline compiles a document (which is often what we want or need)
5. Computations can run in parallel!

Our analysis as a pipeline (3/3)

• [Discussion] Let’s take stock… what else should we do and what’s missing?
• Call renv::init()

Further steps you could take

• Using Git and Github to:
  • Track changes to your code
  • Automate publishing of results (which I think could prove really useful!)
• Put commonly used functions into a package, for example:
  • Earned premium/UPR calculations
• Overkill but use Docker. Further reading: I’m an Actuary: What is Docker and do I need to use it? by Caesar Balona

Conclusion

• At the very least, generate an renv.lock or requirements.txt file
• Write good documentation/audit trails for your code
• Consider using {targets}: not only good for reproducibility, but also an amazing tool all around
• Long-term reproducibility: must use Docker (or some alternative), and maintenance effort is required as well